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Abstract. We study the curvature condition which uniquely charac- 
terizes the hemisphere. In particular, we prove the Min-Oo conjecture 
for hypersurfaces in Euclidean space and hyperbolic space. 



1. Introduction 

There have been many results on characterization of manifolds with non- 
negative scalar curvature. One of the most important theorems is the Posi- 
tive Mass Theorem proved by Schoen-Yau [3 [9], and later by Witten [12] . 
A special case of their theorem tells us that, if (M 3 ,g) is asymptotically 
flat with non-negative scalar curvature, then the ADM mass defined at each 
end is non-negative; furthermore, if the ADM mass is zero for one end, then 
(M 3 ,g) is isometric to Euclidean space. 

For a manifold M with boundary dM, one can ask a similar question: 
Under what conditions on dM is M isometric to a standard model? This 
question is, in fact, related to the concept of quasi-local mass in general 
relativity. Quasi-local mass is a quantity defined on dM which measures 
the energy content of M. A beautiful result of Shi-Tam [10] states that, 
for a three-manifold M of non-negative scalar curvature, if dM has positive 
Gauss curvature, and if the Brown- York mass is zero, i.e. 

(H -H)da = 0, 

dM 

then M is isometric to a domain in Euclidean space. Here H and Hq are the 
mean curvatures of dM induced from M and M 3 , respectively. Miao [6] and 
Hang- Wang [4] also prove some rigidity results on a flat region in Euclidean 
space under different assumptions. 

Besides a flat region in Euclidean space, one can also consider standard 
spheres as the standard model. There have been several attempts to under- 
stand what properties can uniquely characterize the hemisphere S". The 
following conjecture was proposed by Min-Oo in 1995. 
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Min Oo Conjecture. Le M be an n- dimensional compact manifold with 
boundary dM . Assume that M has scalar curvature R > n(n — 1). Further- 
more, assume that dM is isometric to the unit sphere S n_1 , and that dM 
is totally geodesic in M. Then M is isometric to the hemisphere § + . 

While the conjecture is still open, some partial results have been obtained. 
Hang- Wang [U [5] proved the conjecture under the stronger condition that, 
either g is conformal to the standard sphere metric, or the Ricci curvature 
Ric > (n — l)g. Eichmair [lj proved the conjecture for n = 3 by assuming 
Ric > in M and an isoperimetric condition on dM. 

Let us try to understand the assumptions in the Min-Oo conjecture. Ob- 
viously the condition R > n(n — 1) is necessary, because one can otherwise 
perturb the hemisphere at an interior point so that R > n(n — l) — e, for some 
small e > 0, without changing the assumptions on the boundary. However, 
it seems that the assumptions on the boundary can possibly be weakened 
or replaced by other conditions. For example, under the assumption that 
Ric > (n — l)g, Hang- Wang [5] relaxed the totally geodesic condition on 
dM to be the condition that dM is convex in M. 

In this paper, we study the hypersurfaces in Euclidean space and hyper- 
bolic space, and obtain several curvature conditions which characterize the 
hemisphere. We are able to drop the totally geodesic condition on dM, 
and also relax the isometry condition on dM. In particular, we prove the 
Min-Oo conjecture for the hypersurfaces in these non-positive space forms. 
Our method in fact works for a more general situation than the incorpora- 
tion condition we state below. Roughly speaking, the proofs work for any 
compact hypersurface M with boundary dM, as long as the unit n-sphere 
can travel through dM. 

Let M be a smooth hypersurface in M n+1 . Denote by m, i = 1, . . . , n, the 
principle curvatures of M. We define, for each 1 < k < n, the k-th mean 
curvature of M to be 



In particular, 2cj2(k), and o~ n (K) are the mean curvature, the scalar 

curvature, and the Gauss-Kronecker curvature of M, respectively. Our con- 
vention of the mean curvature is that the unit n-sphere has mean curvature 
n with respect to the inward unit normal vector. We say that M is q-convex, 
1 < q < n, if its j-th mean curvature is positive, for all j = 1, ... , q. These 
definitions also make sense for a C 2 hypersurface. 

Let B\ be the open unit ball in the hyperplane W 1 x {0} centered at the 
origin. Denote by C_|_ the upper solid hemicylinder, i.e., 




l<ii<---<«k<n 



C+ = {{x 



i 



X 



) G R n+1 : r < 1 and x n+1 > 0}. 
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Throughout this paper, we denote 

r = ^(x 1 ) 2 + --- + (x™) 2 , for all (x\ . . . , x n ) E M n . 
It is convenient to introduce the following definition. 

Definition 1.1. Let M C M n+1 be a compact hypersurface which is C 2 up 
to the boundary dM. We say that M satisfies the incorporation condition, 
if M satisfies the following three conditions 

(i) dM is diffeomorphic to S n_1 . 

(ii) dM C W 1 x {0}, and B\ is contained in the region enclosed by dM in 
W 1 x {0}. 

(iii) M n C+ = 0. 



The rigidity theorem in Euclidean space is as follows: 

Theorem 1. Let M C M n+1 be a compact C 2 hypersurface with boundary 
dM . Assume that M satisfies the incorporation condition. Suppose for some 
integer 1 < k < n that 

k), (i-i) 

and that M is k-convex if k > 3. Then M is isometric to the hemisphere 
81. 



Note that for k = 2, (II. ID is equivalent to that R > n(n — 1). Also the 
convexity is not required for k = 2. This theorem in particular settles the 
Min-Oo conjecture for the hypersurfaces in Euclidean space. Furthermore, if 
M is a graph of a function, then the incorporation condition can be dropped. 
Here is the result for the scalar curvature. 

Theorem 2. Let u G C 2 (Bi) n C°(Bi), and M u be the graph of u over 
B\ in M n+1 . If M u has induced scalar curvature R > n(n — 1), then M u is 
isometric to the hemisphere S 7 ?. 



Note that the function u here can be prescribed any continuous boundary 
value. Theorem[2]is generalized to the statement for the fc-th mean curvature 
(see Theorem in Section [2j for details.). 

Next, we consider the upper half-space model of the hyperbolic space 
HP +1 . We consider the hypersurfaces satisfying the hyperbolic incorporation 
condition (see Definition 13.51 for details.). This definition is the same as 
Definition II .1^ except that M n+1 is replaced by H n+1 , that the hyperplane 
W 1 x {0} in Definition ll.il (f n|) is replaced by R™ x {1}, and that the cylinder 
C+ in ([m]) is replaced by the following upper solid hemicone. 

e+ = {(x 1 , . . . ,x n+1 ) e M n+1 : x n+1 > r and x n+1 > 1}. 

We obtain similar rigidity results in H ra+1 . 



4 



LAN-HSUAN HUANG AND DAMIN WU 



Theorem 3. Let M C H n+1 be a compact C 2 hypersurface with bound- 
ary dM . Assume that M satisfies the hyperbolic incorporation condition. 
Suppose for some integer 1 < k < n that 

a fc ( K )>2 fc / 2 Q, (1.2) 

and that M is k-convex ifk>3. Then M is isometric to §™. 

Let us remark that when h = 2, the condition (II. 2D is equivalent to 
R > n(n — 1), in view of the Gauss equation (13.11) . The convexity is not 
required for this case. This proves the Min-Oo conjecture in hyperbolic 
space. In particular, when M is graphical, we can relax the hyperbolic 
incorporation condition. The rigidity result for the scalar curvature is as 
follows: 

Theorem 4. Let u £ C 2 (B\) n C l {B\) satisfy that u > in B\ and u = 1 
on dB\. If the graph of u, denoted by M u , has hyperbolic scalar curvature 
R > n(n — 1), then M u is isometric to S™ . 

Here the assumption on u regarding boundary regularity is stronger than 
that in Theorem [2j This is due to the difference of the geometry, which 
we will briefly mention below. As a remark, a more general statement of 
Theorem |4] for the fc-th mean curvature is proved in Section [3] (Theorem [7]). 

An important observation in this paper is that, we can reduce the problem 
of scalar curvature (or of in general) to that of mean curvature, either by 
the Gauss equation, or by the Newton-Maclaurin inequalities (see [3], for 
example). The advantage is that the mean curvature operator is relatively 
easy to handle, especially in Euclidean space. 

The approach is relatively difficult in hyperbolic space. To begin with, 
one must figure out a model geodesic sphere which plays the same role as the 
unit n-sphere in Euclidean space. Besides, the hyperbolic mean curvature 
operator is significantly different from that in Euclidean space. For example, 
the standard comparison principle (see, for example, Gilbarg-Trudinger 
p. 263-267]) does not apply to the hyperbolic mean curvature operator. We 
in fact construct a counter example in Example 13.31 

Our main tool to derive the hyperbolic rigidity theorem is the strong 
maximum principles including the boundary point lemma. These principles 
are known for general quasilinear operators (see, for example, Serrin [11] and 
Pucci-Serrin |7j). But as indicated in Example l3.31 in the actual applications 
one has to be careful for different requirements on the ellipticity, regularity, 
and coefficients. For completeness, we state and prove the strong maximum 
principles in a form we need. 
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The rest of the paper is arranged as follows. In Section O we prove the 
rigidity theorems for the hypersurfaces in Euclidean space. In Section [3l 
we study the geometry of hypersurfaces in hyperbolic space, and prove the 
hyperbolic version of rigidity theorems. Finally we include in the Appendix 
the complete proof of the strong maximum principles for the mean curvature 
operators. 

Acknowledgement. Both the authors would like to thank Professor Rick 
Schoen for many helpful discussions. The second named author would also 
like to thank the warm hospitality of Stanford University, and the support 
of The Ohio State University at Newark. 

2. Hypersurfaces in Euclidean Space 

Let M be a C 2 hypersurface in M n+1 , and A = (A?) be the shape op- 
erator of M with the eigenvalues «j for 1 < i < n . We define the k-th 
mean curvature of M, denoted by (Tk(A) or crfc(K;), to be the k-th symmetric 
polynomial oiik= . . . , K n ), i.e., 

a k (A) = a k (K)= ^2 «ti •••«!*• 

In particular, if M is smooth, <ti(k), 2o"2(k), and a n (K) are the mean cur- 
vature, the scalar curvature, and the Gauss-Kronecker curvature of M, re- 
spectively. Therefore, we can similarly define for a C 2 hypersurface, its 
mean curvature, scalar curvature, and Gauss-Kronecker curvature to be 
<ti(k), 2ct2(k), and cr n (K), respectively. We say a C 2 hypersurface is l-convex, 
1 < I < n, if its j-th mean curvature is positive for all j = 1, . . . , I. 

In this section, we may interchangeably use o~i(A), and Hq to 

denote the mean curvature of a hypersurface. The notation B a stands for 
the open ball in W 1 x {0} of radius a > centered at the origin. Unless 
otherwise indicated, we always denote 

r = ^/(x 1 ) 2 H h (x n ) 2 . 

Let us first proceed to prove Theorem [2l in which the hypersurface is 
graphical. Let u € C 2 {B\) n C (B\) and M u be the graph of u over B\ in 
M n+1 , i.e., 

M u = {{x\...,x n ,x n+1 ) :x n+1 = u(x\...,x n ), for all r < 1}. 

The mean curvature of M u , with respect to the upward unit normal vector, 
is 

Below a simple estimate of the total mean curvature is derived. 
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Proposition 2.1. 



Bi 



Ho(u)dx 



< riVo\(Bi 



Proof. It follows from the divergence theorem that, for any < a < 1, 

Du ■ (y/a) 



Hq(u)oIx 



B„ 



< 



9Ba yTTjDW 
\Du 



dy 



8B a y/l + \Du\ 

n 



- T dy 



< Yol(dB a ) = -Vol(B„) 
a 

Letting a tend to 1 yields the result. 

Proof of Theorem \^ Let us invoke a useful identity: 



n 



* 2 (A) 



+ 



\A\ 



(2) ' n(n-iy 
We denote by ^4 the trace- free part of A, i.e., 

^0 



-61 



n 



□ 



(2.1) 



(2.2) 



The identity (|2.ip follows immediately from substituting (|2.2p into 

^ 2 -2a 2 (^) = |A| 2 . 

Now if 2o"2(A) > n(n — 1) everywhere, we have by (|2.ip that 



n 



\A\* > n 2 . 



n — 1 

Therefore, by continuity, we have either Hq > n everywhere, or Hq < —n 
everywhere. But in view of Proposition 12.11 we obtain in fact an identity 
in either case. Applying (|2.ip again yields that A = 0, i.e., M u is totally 
umbilic with all principal curvatures identically equaling 1. Therefore, M u 
is isometric to the hemisphere . □ 

Next, we would like to prove Theorem [TJ Recall that C + is the upper solid 
hemicylinder given by 

C+ = {(X 1 , . . . ,x n+1 ) e R n+1 : r < 1 and x n+1 > 0}. 

Let M be a C 2 hypersurface with boundary dM. We say M satisfies the 
incorporation condition, if M has the following three conditions 

(i) dM is diffeomorphic to S n_1 . 

(ii) dM C W 1 x {0}, and B\ is contained in the region enclosed by dM in 
W 1 x {0}. 

(iii) M n C + = 0. 
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The following lemma settles the mean curvature case in Theorem [TJ 

Lemma 2.2. Let M C R™ +1 be a compact hypersurface with boundary dM. 
Assume that M satisfies the incorporation condition. If the mean curva- 
ture of M satisfies that \Ho\ > n everywhere, then M is isometric to the 
hemisphere . 



Proof. Without loss of generality, we can assume that Hq > n with respect 
to a non-vanishing unit normal vector field v on M. More precisely, v is 
the inward unit normal vector if we "close up" M by adding the flat region 
enclosed by dM in R n x {0}. Let S(q) be the unit n-sphere in M n+1 centered 
at (0, . . . , 0, q), for each g £ 1, Since M is compact, we can start with a 
very large q so that S(q) has no intersection with M. Then, we continuously 
decreases q until S(q) begins to intersect M. We denote q = qo for this 
moment. 

We assert that if S(qo) is tangent to M at an interior point, then M must 
be a portion of S(qo), and hence, dM = dB\ by the incorporation condition; 
therefore M is isometric to the hemisphere. Indeed, let V = M n S(qo). 
Obviously V is a nonempty closed subset in M. If V ^ M, then there exists 
an interior point p of M such that p € dV. Locally near p, both S(qo) and 
M can be written as graphs over T p M. Note that the mean curvature of 
the graph of S(qo) is equal to n (instead of — n) with respect to the unit 
normal v. This is guaranteed by the incorporation condition ((nj) and (lull) . 
Applying part (pQ) of Theorem [9] yields that M must coincide with S(qo) 
over the small neighborhood of p. This contradicts the assumption p £ dV. 
Hence, V = M and the assertion is proved. 

The assertion will imply that dM n 8B\ ^ 0, and = 0. Suppose that 
dM n dB\ = 0. Then by the incorporation condition S(qo) has to tangent 
to M at some interior point. By the assertion M is a portion of S(qo) and 
dM = dB\. It is a contradiction. Thus, we have qo > 0. If qo > 0, then 
again S(qo) must intersect M at the interior, by the incorporation ([ml) . We 
arrive a contradiction again by the assertion. 

It remains to rule out the case that S(0) intersects M only at dM. For 
any xq £ dM n dB\, we can locally write M and S(0) as graphs ifi and <p 
(with (p > 0) over T X0 C, respectively. Here C is the cylinder 

C = {(x 1 , . . . , x n+1 ) e R n+1 | r = 1}. (2.3) 

We have by the second part of Theorem [9] that 

<9(V> - (f) _ 



g x n+i 



-(x ) > 0. 



Let us also write C locally as the graph ip over T xo C. Because S(0) is 
tangent to C at xq, (p and ip have the same derivatives in the direction of 
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d/dx n+ , and therefore, 

This implies that ?/>(z) < V?(x) for any x = xo — (0, . . . , 0, t) with t > small. 
This holds for any xo G dM n <9-Bi. Hence, there exists a small constant 
5 > such that 

M n C_ 5 = 0, 

in which 

C_ s = {( x \ . . .,x n+1 ) G R n+1 : r < 1 and - 5 < x n+1 < 0}. 

Then, there exists a small constant e > such that S(q) has no intersection 
with M for any —e<q<0. Thus, we can continuously decrease q until 
S(q) is tangent to some interior point of M. We get a contradiction by the 
previous assertion. This completes the proof. □ 

Proof of TheoremUl It remains to show the theorem for k > 2. If k = 2, by 
(|2.ip we have 

\H Q \ > n. 

The result then follows immediately from Lemma 12.21 Assume for some 
k > 3 that M is /c-convex and o~k(K) > (?). Recall Maclaurin's inequality 
states that, 



(2.4) 



for any A = (X\, • • • , A n ) G W 2 with <Jj(\) > for all j = 1, ... , It follows 
that 



0fc(A) 


1/fe 

< 


o-fe-i(A) 


V(fc-i) 

< • 


< ^i(A) 











— n 



ffc(/c) 



> n 



everywhere on M. Thus, the result is implied by Lemma 



□ 



If M is graphical, then the incorporation condition in Theorem [T] can be 
dropped. 

Theorem 5. Let u G C 2 {Bi) D C°{B\), and M u be the graph of u over B\ 
in R n+1 . Assume that M u is k-convex for some integer 3 < k < n, and 



Ok > 



Then M u is isometric to §" . 

Proof of Theorem^ For 3 < k < n, Maclaurin's inequality implies that 

<7jfc(«) 



2o"2(k) > n(n — 1) 



The result then follows from Theorem [21 



l/fc 



> n(n — 1). 



□ 
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3. Hypersurfaces in Hyperbolic Space 



We consider the upper half-space model for the hyperbolic space H n+ 
with the metric x~ +1 5ij. Let M be a C 2 hypersurface in H n+1 and A = {A\) 
be the hyperbolic shape operator. Similar to Euclidean space, we define the 
hyperbolic k-th mean curvature of M to be the k-th. symmetric functions on 
A. As before, M is called k-convex, if <Jj{A) > for all 1 < j < k. Note 
that o~\{A) is equal to the hyperbolic mean curvature H. In contrast to the 
Euclidean case, the scalar curvature of M induced from HP 1 " 1 " 1 is defined to 
be 

R = 2a 2 {A) -n(n-l). (3.1) 

(In the following, we call the induced scalar curvature of a hypersurface in 
H n+1 the hyperbolic scalar curvature.) 

The reason is due to the Gauss equation. More precisely, assume that 
M is a smooth (or at least C 3 ) hypersurface in H n+1 . For any p £ M, 
we choose an orthonormal basis {ej}" =1 of T p M. Denote by Rijki and RijUi 
respectively, the Riemann curvature tensors of H n+1 and M at p with respect 
to {ej}™ =1 . Then, by the Gauss equation, 

Kij — Rijji — Rijji A^Ajj -\- AijAij) (3.2) 

where is the sectional curvature of H n+1 at p. Recall that 

= —l + 5ij, for all 1 < i, j < n. 

Summing (13.21) over all i, j = 1, . . . , n yields that 

- n(n - 1) = R-H 2 + \A\ 2 (3.3) 

where R and H are, respectively, the hyperbolic scalar curvature and mean 
curvature of M. On the other hand, we know that 

2a 2 (A) = H 2 — \A\ 2 . 

This combining (13. 3p yields that 

R = 2a 2 {A) -n(n- 1). 



The following simple proposition relates the hyperbolic scalar curvature 
to the hyperbolic mean curvature. 

Proposition 3.1. Let M be a C 2 hypersurface in H n+ . Denote by A the 
trace- free part of A. Then, 

H Y \ A \ 2 R 

- = TT + 1- (3-4) 

n J n[n — 1) n[n — 1) 

As a consequence, if R> n(n — 1), then 

H > V2n, 
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where the equality holds if and only if M is totally umbilic with all the 
principal curvatures identically equaling y/2. 

Proof. The identity (|3.4p follows immediately from substituting 

i^i 2 = iii 2 + 4 

n z 

into 

R + n(n-l) = 2a 2 (A) = H 2 - \A\ 2 . 

□ 

Let us now look at the graph case. Let u = u(x l , . . . ,x n ) G C 2 {B\) n 
C°(Bi). Throughout this section, we denote for 5 > 0, 

B s = {(x\...,x n ,0)£R n+1 \r<5}, 

and 

r = \x\ = \J (x 1 ) 2 + • • • (x n ) 2 . 

Let M u be the graph of u over B\, i.e., 

M u = {(x\...,x n ,x n+1 ) | x n+1 =u(x\...,x n ), for all r < 1}. 

Notice that if u = 1 on O-Bi, then 9M U with the induced metric from H n+1 
is isometric to the unit (n — l)-sphere S n . The mean curvature of M u , 
with respect to the upward unit normal vector, is 

H(u) = —= +u} , ' . (3.5) 

A geodesic sphere in H n+1 is an Euclidean sphere which is contained in 
HPi+i. The hyperbolic mean curvature of a geodesic sphere is given by 

H= q -n, 
a 

where q is the height of the center and a is the radius. Among all the 
hyperbolic geodesic spheres which pass through dB\ x {1}, the one of radius 
v2 centered at (0, ... , 0, 2) has the maximum mean curvature, which is equal 
to v"2n. We call this geodesic sphere the model sphere. Let 

v = 2 — y2 — r 2 , for all r < 1, 

and M v be the graph of v. Then, M v is the portion of the model sphere. By 
abuse of language, we also refer M v (or v) as the model sphere. 

Proposition 3.2. The model sphere M v , endowed with the induced metric, 
is isometric to the hemisphere S!L . 
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Proof. First, notice that M v is totally umbilic with all the principle curva- 
tures identically equaling y/2. Let {ej}™ =1 be an orthonormal basis under 
which A is diagonalized. Since M v is smooth, we can apply the Gauss equa- 
tion (|3.2[) to obtain that 

— 1 = Kij = Kij — 2, for all 1 < i, j < n and i ^ j, 

where Kij is the sectional curvature of M v . Thus, M v has constant sectional 
curvature 1. Moreover, M v is simply connected, and dM v = dB\ x {1} which 
is isometric to S n_1 . Therefore, we conclude that M v is isometric to Si. □ 

Before proving the theorems, let us remark that the standard comparison 
principle (see, for example, Gilbarg-Trudinger [21 p. 263-267]) does not 
apply to the hyperbolic mean curvature operator H(u). One reason is that 
the second order coefficients in H depend on u. See below for a counter 
example. 

Example 3.3. Let us compare the model sphere v with the following two 
functions: 



ui = 1 and U2 = 1 + e — yl + e 2 — r 2 , for all r < 1 , 
where < e < 1/2. Observe that ui,u 2 G C°°(B{) with 



and that 



JET(iti) = n < V2n = H(v), and H(u 2 ) = n ^ + ^ < V2n. 

Vl + e 2 

Nevertheless, we have 

u\ > v > v>2 on B\. 

Now we would like to show the rigidity theorem for the mean curvature. 

Theorem 6. Let u G C 2 {B{) n C^-Bi) suc/i */iat u = 1 on dB x . If H(u) > 
y2n where H is given by ()3.5p . t/ien 



n = v = 2 — \/2 — r 2 , on B^. 
Proof of Theorem 0. Let us first point out that 

ti > on 5i, 

because, if u achieves a non-positive minimum at yo G then a direct 
computation shows that H(u)(yo) < n. 

Let 



v q (r) = q — y ^ — r 2 , for all r = |x| < min |l, 
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where q > is a constant. Then {v q } defines a family of geodesic spheres, 
of radius qj \[2 centered at (0, . . . , 0, q), whose hyperbolic mean curvatures 
are all equal to \[2n. 

The following two inequalities will be used: First, for all q > \[2 and 
1 + % 

v i\dBi = V 1 ) = 9- V = u\ . (3.6) 



Second, 



v q{ r ) > r ) f° r all g > and all r < q/V2. (3-7) 



We start with a very large q so that there is no intersection of the graphs 
of v q and u. Then, we continuously decrease q until ultimately the graph of 
v q begins to intersect the graph of u. Denote by q = qo for this moment. 
Observe that go > 2, since v q , with q = 2, is exactly the model sphere v 
which intersects the graph of u at least on dB\ . 

We assert that qo = 2. Suppose that qo > 2. Then v qo E C°°(jBi), and 
v qo > u on B\ by the above construction. In view of (|3.6p . v qo must be equal 
to u at some interior point in B\. Applying Theorem [9] ([T|) with Q = H and 
V = B\ yields that 

Vg = u on B\. 
This contradicts with ()3.6f) . The assertion is proved. 



Thus, we have 

v qo = v = 2 — \Jl — r 2 , for all r < 1. 

Clearly, v £ C°°(Bi), and d > m on Bi. Note that if v = u at some interior 
point of -E>i. Again by Theorem [9] we have u = v on B\. Therefore, to prove 
this theorem, it suffices to rule out the remaining case, i.e., 

v(x) > u{x) for all x in Bi. (3-8) 



Assume that ()3.8p holds. We claim that there exists a sufficiently small 
constant e > such that for any 2 > q > 2 — e, 



/ (? 2 

g-W- — r 2 > u on B±. 

Deferring its proof, we first proceed to exclude (|3.8p : By the claim we can 
find a g E (2 — e, 2) so that the graph of t> 9 lies completely above the graph of 
u over B\. Then, we can move the graph of v q downward, by continuously 
decreasing q. Note that eventually the graph of v q must intersect the graph 
of u, since u(0) > and 

v q (0) = minvq = ( 1 — J g — > 0, as g — > + . 

Bl V v2/ 
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We denote by q\ for the moment that the graph of v qi begins to intersect 
the graph of u. There are two cases: 

Case 1°: Suppose that 2 > q x > y/2. Then v qi £ C°°(Bi) n C°{B X ) 
and v qi > u on B\. Because of (|3.6p . v gi has to equal u at some interior 
point of B\ . By Theorem [9] (pQ) , we obtain that v qi = u on B\ , which is a 
contradiction. 

Case 2°: Suppose that y/2 > qi > 0. Then v qi £ C°°(B T ) n C°(B T ) and 
v qi > u on i? T , where r = q\j\pi < 1. Note that 

— — (r) = — = — > +oo, as r -> r. 

This implies that the graph of v 9l cannot touch the graph of u over a point 
on dB T , for otherwise u would not be in C 2 {Bi). Thus, v qi = u at some 
interior point of B T . However, applying Theorem [9] ([T]) yields that v qi = u 
on B T ; this again violates the assumption that u £ C 2 (Bi). Therefore, 
combining the two cases, we have ruled out (|3.8p . with the aid of the claim. 

Let us now settle the claim. A key observation is that there is a small 
constant 5 > such that 

u(x) < \x\ = r, for all 1 — 8 < \x\ < 1. (3-9) 

In fact, by (|3.8p and applying Theorem [9] ([2]) yields that, 

< (Du — Dv)(xq) ■ xq = (Du — Dr)(xo) ■ xo, for each xq € dB±, 

where we use the fact that 

Dv(xq) = Dt{xq) for all xq £ dB\. 

Then by continuity, there exists a sufficiently small 5 > such that 

D(u - r)(x) • -A- > 0, for all 1 - 6 < \x\ < 1. 

\x\ 

Since u £ C 1 (-Bi), we have by the mean value theorem that 
u(x) — r(x) < 0, for all 1 — 8 < \x\ < 1. 
This proves (|3.9|) . 

Let 

d = — mm{v(x) — u(x) : \x\ < 1 — 8}. 
Then d > 0, because of (|3.8p . On the other hand, by the uniform continuity, 
sup |t; - v q \ < C(2 - q), for all (2 + y/2)/2 < q < 2. 

where C > is a constant independent of q and 8. Now let 
e = min{d/C, (2 - V2)/2} > 0. 
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We then have for any 2 — e < q < 2, 

v q > v — Ce > v — d > v — (v — u) = u, for all \x\ < 1 — 8. 

Moreover, combining (|3.7p with (|3.9p gives that 

v q (x) > \x\ > u(x), for all 1 — 5 < \x\ < 1. 

Therefore, we conclude that for each 2 — e < q < 2, 

> for all \x\ < 1. 

This proves the claim, and hence, the proof is completed. □ 

Remark 3.4. As pointed out by Rick Schoen, if the function u in Theorem[6] 
satisfies in addition that du/dr = dv/dr on the boundary (which holds if 
dM is totally geodesic in M), then (|3.8|) can be immediately ruled out by 
Theorem M ©• 

Theorem |4] then follows immediately from Proposition 13.11 and 13.21 and 
Theorem [H There is also a /c-convex version of the rigidity theorem, which 
also follows from Proposition 13. II and 13. 21 Theorem [6j together with Maclau- 
rin's inequality (|2,4p . 

Theorem 7. Let u G C 2 (i?i) D C 1 (-Bi) satisfy that u > in Bi and u = 1 
on 92?i. Xei M u 6e the graph of u over B\. Assume for some 1 < k < n 
that 

a fc (.)>2 fc / 2 Q, 
and that M u is k-convex if k > 3, then M u is isometric to S™. 

Next, we generalize the rigidity results to a hypersurface in H n+1 . Sim- 
ilar to the Euclidean case, we will introduce the hyperbolic incorporation 
condition. Let <£+ be the upper solid hemicone, i.e., 

£ + = {(x 1 , . . . G M n+1 : x n+l > r and x n+1 > 1}. 

Definition 3.5. Let M C H n+1 be a compact C 2 hypersurface with bound- 
ary dM. M is said to have the hyperbolic incorporation condition, if M 
satisfies the following three conditions: 

(i) The boundary dM is diffeomorphic to S n . 

(ii) dM C M. n x {1}, and B\ is contained in the region enclosed by dM in 
W 1 x {1}. 

(iii) M n <£+ = 0. 

The rigidity theorem for hyper surf aces is as follows: 

Theorem 8. Let M C H n+1 be a compact C 2 hypersurface with boundary 
dM . Assume that M satisfies the hyperbolic incorporation condition. If M 
has hyperbolic mean curvature H > V2n, then M is isometric to S!f ■ 



RIGIDITY THEOREM ON HEMISPHERES 



15 



Proof of Theorem 0. The proof is similar to that of Theorem [H so we only 
point out the difference here. We use the following family of functions, 

r 

v q (r) = q — -w — — r 2 , for all r < q/v2, 

where q > serves as a parameter. Denote by S(q) the graph of v q . Again 
we start with a very large q so that S(q) has no intersection with M . Then 
we move S(q) downward, by continuously decreasing q, until q = qo when 
S(q) begins to intersect M. 

Observe that S(qo) n M is contained in the interior of S(qo). To see this, 
notice the fact that 

dS(q ) = {(x\ . . . : r = q /V2,x n+1 = q } C S(3q ). 

Thus, if there exists an x £ dS(qo) D M, then S(3qo) should also intersect 
M at x. This contradicts our choice of go- 

For any y £ S(qo)C\M, the observation enables us to locally write M near 
y as a graph over a small ball V C B% . The (jm|) of incorporation condition 
assures that the local graph has the right sign for its mean curvature. Thus, 
similar to the proof of Theorem [6J we can apply Theorem [9] to obtain that 
qo = 2, and that M is either isometric to , or S{2) only intersects M at 
dM. It remains to rule out the latter case. The process goes the same as 
that of Theorem [6l except that the Case 2° here is excluded by virtue of the 
above observation. The difference is that a hypersurface can be vertical at 
the interior of B\, in contrast to a graph. □ 

Finally, Theorem [3] follows as a corollary of Theorem [8] and Maclaurin's 
inequality. 



4. Appendix: Strong maximum principles. 



Throughout the appendix, we denote by B r the open ball in M. n x {0} of 
radius r centered at the origin, and denote B = B\ for simplicity. A function 
with subscripts stands for the derivatives of the function. For example, 

du d 2 u 

m = 

Consider 



* dx l ' y dx l xi 



Q{u) = ^2 a ij (u, Du)uij + b(Du) in B, 
for all u G C 2 (B) n C°(B). Here 
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and 

b(p) = ? == , for all p = (p\ . . . ,p n ) G W 1 , 

v 1 + \p\ 

in which bo is a constant, and z = z(t) is a smooth function defined on a 
domain in M. In particular, Q is the Euclidean mean curvature operator 
Hq if bo = and z = 1; Q is the hyperbolic mean curvature operator H if 
bo = n and = t. 

Theorem 9. Let p,ip G C 2 (5) n C°(5) wii/t z(^) >0on5. Lei V 6e an 
open ball in B (V could be B itself) such that 

Q(np) < Q(ip), and > ^, in V. 

^ If ip = ip at some interior point of V , then 

ip = ip on y. 

(2) If<peC 2 (V), ^^(^nC 1 ^), tp>ipinV, and ip = ip at some 
xq G dV, then 

D(ip - (p)(x Q ) • V > 0, 
where r] is the outward unit normal vector at xq to dV . 

Remark 4.1. Part ([2]) of the theorem is the boundary point lemma for 
quasilinear operators. The regularity of <p and ip can be replaced by that 
at least one of <p and ip belongs to C 2 (V) H C 1 ' 1 (V r ), while the other is in 
C 2 (V) n C^ l {V) and at which z > 0. 

The proof of Theorem [9] is based on Hopf 's strong maximum principle 
and boundary point lemma. Let V be the open ball in B, and L be the 
linear operator given by 

n n 

Lh= a ij hij + ^2 bifl i + ch in V > ( 41 ) 

i,j=l i=l 

for all h G C 2 (F). Assume that the coefficient matrix (a l i) is everywhere 
positive definite in V, and that a 1 -?, 6*, and c are continuous in 1/ for all 
1 < ijj < n. A special case of Hopf's strong maximum principle can be 
stated as follows: 

Lemma 4.2. Let L be the operator given by (|4.ip . and h G C 2 (V r ) sucm that 
h < and L/i > in V. If h = at some interior point of V , then h = 
on V". 

Lemma 14.21 follows, in turn, from the following boundary point lemma 
due to Hopf. 
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Lemma 4.3. Let L be the operator given by (14. IB . and U be an open ball 
in V such that a %3 , b % , c G C (U) for all 1 < i,j < n, and that there exists 
a positive constant such that 

n 

aij tei > e \Z\ 2 ' ( 4 - 2 ) 

for allxeU and £ = ...,£„) G M n . Le* h G C 2 (?7) n C 1 ^) «ucA i/iai 
L/i > in U. Suppose that for some xq G dU, 

h(x ) = > h(x), for all x G U. 

Then, 

Dh(x ) ■ fi > 0, 
where [i is the outward unit normal vector to U at xq. 

Proof of Lemma \4-3[ We assume without loss of generality that U = B$ for 
some < 5 < 1. Define 

u.(x) =e" A|:!;|2 -e" A52 , forallxG£ 5 , 

where A > is a constant yet to be determined. Notice that 

(L — \c\)w 



> e 



-AM 2 



4A 2 aijx * x 3 ~ 2A [J2 bixi + Yl °* J' d c l " c ) 

i,j=l \i=l i=l / 

> e- A|x|2 [4A 2 0|x| 2 - 2AC(|x| + 1) - C] . 



Here 6 > is given by (|4.2p . and C > is a constant depending on the 
C°(y)-norms of a 11 , b l , and c. Now consider the annulus A = B$\B$/2- We 
can choose a sufficiently large constant A = A(#, S, C) such that 

(L - |c|)u> > on A. (4.3) 

Since /i(xo) = > h(x) on dB§/ 2 , there is a constant e > such that 

h(x ) = > /i(x) + eu>(x), (4.4) 

for all x G dB$j 2 - Note that (|4.4p also holds on dB$, where w is identically 
zero. On the other hand, we have by (|4.3I) that 

(L — \c\)(h + ew) > —\c\h > on A 

It follows from the usual maximum principle that 

h + ew < on A 

But /i(xo) + ew(xo) = 0. Taking the normal derivative at xo yields that 

du . . dw . . 
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Thus, 



|(x )>- e ^(xo) = 2A^ 2 >0. 



□ 

Proof of Lemma \4-°A Let 

E = {x G V | h(x) = 0}. 

Then £7 is relatively closed in V. By the assumption E is nonempty. We 
need to show that E = V. Suppose not. We can then choose a point 
y £ V \ E such that 

d(y,E)<d(y,dV)/2. 
Consider the largest open ball U C V \ E centered at y. Then, by the 
construction U C V and dU must intersect E at some point xq. Thus, this 
implies that a ij , b\ c £ C°(U) for all i,j = l,...,n, that (|3~2|) holds for 
some constant 9 > depending on C7, and that /i £ C 2 (U) and 

/i(a;o) = > for all x £ U. 

Applying Lemma 14.31 yields that 

£>%o) + 0. 

This is a contradiction, since xq is an interior maximum point of h in V. □ 
Proof of Theorem 0. Note that 

Q(V0 - Q(<p) 

n n 

= J2 5 y (VMty)(V'ij - tpij) + <Pij[a ij (i>,Di;) - a ij (^,Dtp)] 

i,j=l i,j=l 
n 

+ b(D^) - b(Dip) + Vij Dtp) - a ij (tp, Dtp)] . 

Let h = ip — tp. We can rewrite 

n n 

< Q(if>) - Q(tp) = a ij hij + b% + ch = Lh, 

i,j = l i=l 



where 



a *3 = a l3 (i>,Dip), 

"1 fi^hn 



> l = V tp lm / *^,tDi> + (l-t)Dtp)dt 
f 1 db 

+ J — i {W^ + {l-t)Dtp)dt, 

n rl Qoj- m 

c= Y tpim J -r^-(ti/> + (l-t)tp,Dtp)dt, 



Lm=l 
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for all i,j = 1, . . . , n. Then since tp, if) G C 2 (B), a l i , b % , c are continuous in 
B for all i,j = 1, . . . ,n. Furthermore, we have 

for all x G -B and £ G M n \ {0}. Thus, the first part of Theorem [9] follows 
immediately from Lemma 14.21 For the second part, since <p G C 2 (V) and 
ip G C l iV), we have ,b % ,c G C°(V") for all 1 < i,j < n. Furthermore, 
(|4.2|) holds as we can take 

9 - T (i + i^l 2 ) 3 / 2 

Now applying Lemma 14.31 with U = V yields the result. □ 
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